Oh yeah. You don't put your headset microphone? Okay. Yeah. Yeah. Yeah, I have actually yeah, the we can see the report here and the demo, and the goal of uh this meeting is to present the video shot detector to Olivier, okay. Okay. So yeah, it was my project um during my internship at IDIAP under the supervision of Jean-Marc. And as you saw as you saw in the email I sent uh okay. I didn't send this video is the new one. I just um uh processed from your images. Yeah. But uh let's let's start by the video from Friday. CINETIS demo one. Yeah. So this was uh Okay. Yeah, here on the left you have different things, but uh as applied So this detector uh detected the thirty shots, and if you click on uh plus button you can expand it and see the key frames that are in fact uh five extracted frames from the shot. But I think th uh as it is, it's t extracted uh without uh using sub-shots, uh it's simply uh five down-sam yeah, down-sampling. Um so I we can see the video actually, uh the entire video, can open with Real Player. I hope it will work. So we don't have the the sound, and it won't be grabbed the video won't uh I don't know if uh Okay, it's not good quality. Oh yeah, yeah, yeah. No no no no no no. It's in Real Media, so it's very compressed. Uh okay. Uh ten yeah. And then there's um Okay. Yes. And this one, yeah, it's a dissolve it's a dissolve, so it doesn't detect yeah, when I saw the M_ with M_ Player, it's a dissolve, yeah. Yeah. Okay. Yeah. I don't know if this one if we see was correctly uh, you see? Yeah. Yeah. But there is a moon. But okay, because there is no motion in fact between the two shots. Yeah, okay, okay. The cow? Yeah. Mm-hmm. Yeah, maybe to explain this method was using uh a simple uh uh distance between uh consecutive histograms, and we didn't use motion features at all for that, that's why i we Yeah, okay. Enough Yeah. Yeah. With a sliding window. I can show you uh Yeah. Yeah. I don't know if we c we want to make uh use the whiteboard or here it's uh it was uh to dis it's infra only the this part about uh shot boundary detection is in French unfortunately, but uh the this image shows the results using motion when you s use the motion features, and here on the right is when using the histogram uh distance. And the the motion features can better show what's going on when there is such a difficult uh transition, while with the histogram um, yeah, it just shows that there is a shot uh change at the end, but even for dissolves, I don't have examples here, but um there is not a strict uh transition. Uh it's continuous for dissolves. Dissolves is uh yeah. Yeah. No no, now it's uh well for the video you saw it wa it's Bhattacharya. Yeah. Okay, set the beginning. So there is the idea is having a sliding window. Um we have already all the data that is on the graph. And then to detect uh the the the the changes, we we do a calculation um with points under this sliding window within this sliding window. And uh we calculate so what do we here? Um In fa we want to detect that uh this points in the middle is higher than uh Th Exactly, yeah. Alpha. Alpha times the maximum of the other ones. The c But if you are in the maximum already and at this point we check simply if uh this value is alpha times the the max of the other points, let's say these points here. Um i maybe for chi it's it six but uh, yeah, but uh for Bhattacharya it's for the tests I did it two point four. Yeah. Yeah. Okay. No Mm. Uh i in the the first shot? Yeah, I saw that it was detected as a shot, yeah. You want to see now? Okay. But he didn't okay. It is uh okay. This is IDIAP laptop, so you can u okay. So here in fact after twelve um I don't know if it's I think it's twelve frames. Yeah. Oh yeah, maybe you're burning, but Oh, there is no ha ha. I didn't uh ha I didn't have time to the the Real Media. But um yeah, it's complicated. Yeah, here we can see that, yeah. At two point four. Mm. And then there is translation on the right, yeah. Mm mm mm. I can see uh there is another programme that can see Uh the N_ uh It's I think it's f there there are five normally. If we put expand all Yeah. The other one I had problems converting uh uh the video into D_V_X_, and then uh so I used the programme from uh software from Sebastien, P_ P_A_P_M_ to A_V_I_. Because yeah, one thing uh maybe we will talk about that is the library uh, it's uh it doesn't uh compile anymore on Debian machines and, yeah, some maybe some work has to be done on this. I don't know why this uh image doesn't appear, this is the um um the frame that is uh uh extracted to be the shot key frame here. This one is this one, the middle one. So I don't know what why it doesn't appear, but it just uh a simple problem in the get frame. But what I can show you now is a little programme if we want to be um very precise about so it's ma get frames Okay, C_G_I_ here. And we were interested in around what di do you want? Around six, seven seconds? Okay. Uh if we do next. Oh. Yeah. There is something wrong. So it detects Yeah, in fact, yeah, that's why I said it's it's a frame number. Uh w I don't know why, yeah yeah yeah. It's a frame number. This is a frame number, this is in second, but I I kept the frame number. It's twelve frames, zero seconds. Yeah. Yeah. Yeah. So if we do every second uh I d the middle frame is uh this one thousand Okay. Do you thi it's another kind of browsing page, but uh to be more precise about uh to to check maybe the this yeah, exactly, yeah, yeah. Anyway, you go back to one frame around f one sixty for instance, I don't know. And and there is yeah, that's it, you know. Prev Okay. You see that there is uh okay, it's not a shot change, it's a motion actually here. Yeah, yeah, yeah, yeah, yeah. Yeah, I s yeah. Yeah. When you have yeah, the the goal results uh then yeah, in fact it's not a shot change. Yeah. Yeah. So this would ma be made uh manually actually. When you say from this shot to this shot is using a graphical user interface someone who would do manually for each video. Method, yeah. Yeah, mm. Yeah. Ah, okay. Yeah, yeah, yeah. Mm. Okay. This is the one in the D_V_D_? Maybe okay, we can uh see a little bit if you want, to play the Well no, it's okay. Okay, uh in fact I didn't understand that, but now I understand better that CINETIS wants this kind of method to help them to restore and to apply the algori some algorithms to to to make a better movie, okay. It's not for the client, for the user who will uh maybe okay. Okay. Also but okay. Okay. Mm-hmm. Mm-hmm. Mm-hmm. Yeah, uh okay. Okay. Ah, okay, yeah yeah yeah. Because he did the movie, so he wants t okay, to archive, he wants to okay. This is yeah, another mm. Yeah. Yeah. Yeah. Okay. Mm-hmm. And this could be done by through internet. Mm. Mm-hmm. From the client point of view and from the preparation of the movie D_V_D_, mm. Okay, I will play, yeah. We had an arrow here or what? It's Th If I go to menu there is something? Do you know what this is? No, old menu uh there is no menu in the D_V_D_? Okay. Okay. Okay. So this is the same video a the the A_V_I_ number one? And on the on the D_V_D_ I saw that there are some other stuff. I dunno. This one? The same. Okay. Oops. Ah, okay. Uh. There are two files. Okay, I understand. Okay. Run the okay. Um Yeah, almost. I would maybe a less a little bit less than uh a bit more than real time maybe. I will have to check, but if it's five minutes videos, it's like done in yeah, quickly. Okay. Components open C_V_. Yeah. Yeah. Okay. Open C_ yeah. Yeah. So that we'll use the whiteboard. So in fact, yeah, the interface is quite is quite simple to use. Um for instance you see um on the uh on the U_R_L_ I put okay, I I disco there is M_M_M_ dot IDIAP dot C_H_ slash V_C_G_I_ bin slash video browser. Okay, uh slash video browser three frames, three frames dot H_T_M_L_. And then this C_G_I_ programme ask the mm parameter C_ video and M_ equals CINETIS okay, demo one. In fact um I just explain how to put the data on M_M_M_, but it's quite simple. When you log into M_M_M_ as with a C_ C_ S_S_H_ for instance, um so you just put it's working like this, M_M_M_ data video browser. Uh okay, slash. And then This is the name of the new directory you will create. Demo one. And there stands there is couple of things well it's uh this name so I will say this is video name, so video name dot A_V_I_, you just because we need the A_V_I_ so that we can uh Yeah, we can do the the get frame Otherwise it's not necessary. Oh, uh f and also for the Yeah, and this is no no no, uh this is done on the fly. With get frame. So you need that, you need uh the X_M_L_ v so it's video name And I could data dot X_M_L_, but in fact it's just a simple convention, I w could have put nothing, but I just put data. The it's a convention. Data, and then video name. Okay, it's R_M_ nine dot R_M_ for other stuff. But in fact, yeah, when I think now, we could have uh Only with uh extension you can you can see. The name is no of in fact yeah, is is yeah. Yeah, is the same as the directory. It could have been video naming also, but it's the same as a directory. And that's it actually. When you you u you do that I did the same for kayak you say kayak in fact now it's kayak, and then you can see. And this output yeah, as your this output uh this file is an output from the mm the software, the shot boundary detection software. It's a very simple format, yeah. Yeah, uh actually we Uh yeah, I don't yeah, we cannot see. It's, yeah, it's uh tha video, scene, shot, it's hierarchical in X_M_L_ and key frame time stamp and the and from the time stamp and the video location he knows how to get the frame. Each of these frames. And if we do in fact I don't know yeah, it's Mozilla Firefox view page source okay, view frame source. And every frame is stored into uh a b a backup in fact every frame um it's M_M_M_ who manage this automatically, so that um when uh you ask for this frame, in fact we can okay, I just view frame source When I view image, when I say view image, when I say view image on this frame, up we go there and this is um in fact it's a Perl programme that outputs in fa instead of outputting H_T_M_L_ or text, it outputs uh J_ PEG. From the video with a time stamp it outputs a J_ PEG. This is quite uh nice. And Yeah. And this image is i this image is output uh outputted to the client. Yeah, exactly, yeah, yeah. That's what I did uh here. In the other application, yeah. Yeah, I dunno. Yeah, here. I f there are, yeah, the M_M_M_ developers, so I'm part of that, but uh we could we could uh yeah, it's not a problem. Um I wanted to saw to see to do s say something else. Um oh yeah, and the images are sent to your browser, the client, to this browser, but also kept in uh a directory in uh on M_M_M_. Exactly, a cache. I say backup, but it's a cache. Um that's that's how it works. No no no, it's okay. Um for the cache uh We have plenty of space on M_M_M_, so the this is not a problem ah, yeah for a new window. Yeah, exactly, yeah. And it detects okay. Yeah. So you go inside the get frame function, you see where it i it saves it, and then there is a special naming convention, and you can remove the but i it's c yeah, it's true. No. Because there it's in cache, so that if I put another video with the same name, they th they will happen again. They will uh appear, this frames will appear. Yeah, yeah ex if you have the same decomposition, but But uh yeah, on M_M_M_ f now we don't have such many videos, so. But if you put yeah, in another directory, uh then it would work or or a new naming convention with uh special uh, yeah, code uh. Yeah, M_M_M_ cache. Yeah. But it it can go in the browser cache also. The first time is done online. Yeah, you can see here, yeah. Uh the first time is done o You know, you see that. Well here it's it was cache, because I already pressed preview, but now it's on the fly. So it's quite quick. This. And then if you come yeah. Because the images are already generated. And maybe they are on the cache of this browser, this computer. So okay. That's why it's yeah, it was uh used with uh yeah, Sebastien uh did the C_ function, and uh, yeah, and uh the the is calling this uh C_ uh executable and it was used with M_M_M_ server with Pierre, a and the beginning when, yeah. Um yeah, yeah. Oh, the movies or In fact uh we we we could, you know okay, you know all these graphical user interface that you see uh okay. In fact okay. Mm yes. Yes, we could do that. We could put uh CINETIS slash here, and then a CINETIS slash is the sa yeah, it works. But the uh th the point is this interface is uh I think six or seven files only, X_M_L_ uh well it's per C_G_I_, that output, X_M_L_ and X_S_L_. So it's it's not a big deal to to to make a new, like a CINETIS browser or, you know, video browser. Yeah, we w we copy all the files and there is uh just one um one parameter, one uh string, that we need to know where this data is, and we could pr create a M_M_M_ data video browser uh CINETIS or something else. Y it's uh ex extensible and uh is Yeah. Yeah, exactly, yeah. Yeah. Yeah, but i it's not uh it was uh, you know, thi this is working, the expand all but uh this is uh Yeah, to do list or something. On a video annotation, on a video structure co correction. But i it's not easy on the web with H_ H_T_M_L_ pages to to do uh such corrections like uh video structure corrections for instance. For that you need it's possible maybe uh on interfaces like Flash or Java, uh but still uh i i i there is uh it takes bas more work and but i yeah yeah, I mean it's good good point. Well uh you we can make for for instance for T_S_R_ T_S_R_ uh videos. Alessandro uh asked me to do the same thing, and we I put uh we had uh like a password convention. So this is possible to make for these videos. So it's not uh Well if yeah, it is there is no possible productions for these videos. Maybe what I say I'm very tired. But uh if po yeah, it's demo and uh one can say you have to guess the name to but uh yeah. Bu y yeah, a good point uh about password protection issues uh or the privacy issues uh Okay. Uh ha yeah, all the framework is ready with a T_R_S_ T_S_R_ videos, so. Usually what we do sometimes uh okay, just to say something about password protections. Sometimes Instead of having um a purely secure directory we p we we put a co code number like uh so that it's quite impossible to guess instead of video m uh name equals kayak then we would have uh zero six seven uh five or, you know, two. And then this directory from outside we have to take care that okay, not only the video name this U_R_L_ is not accessible, but also we should take care that in fact M_M_M_ it's in M_M_M_ data video browser and this is accessible from outside. We should take care that this directory or in fact this directory is not listable, you know, you cannot list from outside uh with uh F_T_P_ or H_T_T_P_. Yeah, simple things with privacy issues, but uh i there are some methods and everything is possible to make things private. I yeah, I don't know if you want to add anything, but I would say that for the C_ code uh, yeah, I I I just want would like to stress something, is that um, yeah, if you want want to use the new versions, etcetera, I think uh we need to go a little bit through the code and I would be happy to to to help to to for instance the classes about video structure, etcetera, to put them yeah. And Yeah, in fact it's not there is no tosh i it was called vision lib to read uh to to extract one frame each frame of a video, you know. So it wa from Sebastien it was in the tosh library, but uh it's nothing w to do with tosh and it's using uh uh the open C_V_ and uh some classes from uh Jean-Marc uh for, yeah, Bhattacharya distance, etcetera. Image processing. Image processing and stuff. We can forget tosh yeah. Yeah. Um so it's four thirty, so we need one hour, it's it's good. Just if you want yeah. This is the parameters of the of the C_ programme but uh usage. Video in, okay. Uh the output file X_M_L_. Output file dist, it's the distance um between each histogram, so that if the X_M_L_ uh you feel there is not enough um uh shots, you can instead of reprocessing the video, you you reprocess this file. And also some other stuff s for instance alpha threshold we talked about is set to two point four and uh yeah, so the uh the the step we always use one and it should be one, because otherwise it's stuck between consecutive frames. Yeah, but uh it doesn't w yeah, it it's better to to keep it one. Yeah. So detection latency is about the um it's half of the windows tha yeah, sliding window,. Option for key frame extraction, okay. Uh sub-threshold, yeah. Leave it to one, yeah. Uh yeah. Maybe, I I will uh I will give you this file in uh as a rhy it's a kind of rhythmia, yeah. I just for preparing this uh nice discussion. M um I'm quite happy to to to do that i it's good when you do uh something that there is someone using it. But uh but At least, yeah, with the interface it's quite funny to to see um d that the q quickly you can do something in in, yeah, in thirty t if the video i ten is ten minutes, in thirty minutes 'cause it's no not a lot of things to change. Or even it can do it uh it can we can do it quite automatically. It can be done, yeah. Yeah. Yeah. Yeah. Oh yeah, it So the Real Media is uh, yeah, a parallel thing that from the A_V_I_ you convert it into Real Media, and it is done on Windows. Uh Yeah. Mm there is a a the free one is okay. You can use the free we have a license here at IDIAP, but uh the if you want to pay, you pay for options such as uh resize or cropping, but in fact maybe we don't need that, and uh it's called Elix producer. Yeah, I forgot yeah, yeah uh, Elix producer and and that's it. You can you can i it it i Yes. Yeah. Uh yeah. And yeah, with Real Media Well this is you can buy or you can order for free on internet. No, no no no, for the Real Media. And um you didn't see, but the Re the Real Media um is called via a real server, so it's R_T_S_P_ the connection, and i it's streaming video streaming, so it's not H_T_T_P_. So that it can play uh smoothly and but we are still uh investigating for the video formats, because um i it's not supported for instance uh uh in Java, in Java applications uh etcetera. Real Media. Yeah. Real Media, so there are many issues about that, but yeah. It's another problem. Yeah. Yeah, that was a single video structuring, but it's uh other level uh it can uh it's quite uh it's researched, yeah. Yeah. Yeah, last thing uh this is a kind of uh X_M_L_ that is outputted, so so it's uh the structure is such as uh vi video segmentation with uh videos in begin and then for each shot you have an I_D_ well this is the terms from M_ PEG seven convention, but anyway. Shot begin and end. Uh a sh the shot key frame that is the middle here. Uh sub-shot Yeah, you're right. Twelve frames. And in fact that's what I output in uh on the yeah, I think y on the interface. And there is only one sub-shot because uh w yeah. There was there was problems with detecting the sub-shots, etcetera. But uh yeah well. Yeah, I tr well uh not with this one, but yeah, I tried to code some stuff uh it but uh I'm not so confident in the way uh it is uh coded. Yeah. Yeah. But we yeah, even we should uh for instance for the videos uh for the spectral clustering, for the clustering. Here uh I store the consecutive the distance between consecutive histograms, but we could uh store the matrix that's yeah. And then uh yeah. And I think i there are many possible things with uh spectral clustering, but but the uh with um there is one parameter that is very c wha I can't remember, what is the parameter uh Yeah. No no no no, the in the the gap to find the the good number of clusters is so sensitive uh Yeah, anyway. It's it's a bit later in the in the report. So anytimes, yeah, I would be happy to to to help you with that. Yeah, yeah, you can we can go.